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ABSTRACT 

The purpose of this study was to examine how 
end-users participate in the evaluation of multimedia programs in 
North America. Questionnaires were sent to over 100 projects (26 
returned) focusing on instructional objectives. Results are discussed 
in tf of graphics, the user/machine interface, costs, program 
desig.i. quality indicators, such as help and feedback availability 
and user control, team production, end-user participation, and the 
overall product quality, The following conclusions were made: (1) 50 
percent of the educatiohal programs were "information distribution 
programs"; (2) the hypermedia model is widely suited; (3) there was 
continuous evaluation during the production of the program and 
participation of end users in this evaluation; (A) an objective 
mechanism for quality control is not systematically used; (5) the 
multimedia programs depend heavily on the text; (6) the mouse is the 
most common interface for the user; and (7) the multimedia conception 
of communication is being used at a high and costly level by 
developers and at a low level by professors, due to authoring tools, 
and because there is no consideration of the personal costs, the 
differences between budgets have no relation to the quality of 
results. (Contains 25 references.) (AEF) 
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During these years, several people have directed to me, asking "how have I to 
produce a multimedia programme for learning purposes?". And, usually, they 
waited an answer as "put this kind of graphics, in this place, with this sound,..." I 
think that a book as "Multimedia Programmes Production guidelines" would be 
a success of selling. 

In 1994 I arrived to conclusion that the question was correct, but the answer that 
everyone could find in several books were wrong: the bad answer for the good 
question. This is not a new situation. During a travel by plane I asked to Dr. 
David V. Williams (Ithaca College) about the tools that they used to develop 
multimedia software, and he answered me: "Oh, it is simple, the brain'.". This 
time I was waiting ihe bad answer ("Toolbook, AuthorWare, HyperCard...") for a 
good question. And he gave me the good answer. 

So, which is the good answer to the questions about "Multimedia Production 
guidelines"? 1 arrived in 1994 to the conclusion that the answer was not in the 
specific resources or techniques used but in the general role of Evaluation (you 
can in some way understand it as "Quality control") and the role of End-users in 
the production process. Later, during 1995, the European Community has 
introduced these aspects as key issues in the funded project. 

So, between January and December of 1994 Dr. Lauran Sandals and me, we 
interviewed several people involved in Multimedia programmes production 
processes in NorthAmerica. Although we sent questionnaires to more than 100 
projects, we received only feed-back from 26. The length and, perhaps, the kind of 
some questions, discouraged for other people to answer these questions. With 
these answers, some of the initial objectives related with the definition of 
Multimedia Programmes profiles were given up. Here we will explain only some 
conclusion related with the first aim of this research: "To develop a revision of 
Multimedia projects developing processes in Educational context, in North- 
America (US and Canada), relating Evaluation and end users participation 
techniques with the global quality of these products. 



1. Theoretical Context 
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The research about media in Education has traditionally been more involved in 
resources application than in production aspects. Thus, we have some interesting 
works as the revisions of Clark (1983) about the benefits of new media from the 
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old ones. Or the meta-analysis of Kulik (Kulik et Alt, 1980) when they found an 
effect size of 051 between with and without media teaching programmes, effect 
that was reduced to 0.13 when both control and experimental groups had the 
same professor. Or the revision of Bosco (1986) when he found that, while the 
100% of the research without statistics tests reported benefits from the use of 
media, only the 50%, more or less, reported when statistics tests were in use. 
Other aspects have been considered and from different perspectives in the 
research on Instructional Media as Clark (1991), relating to cognitive issues and, 
also, to economic issues. 

Inside the Production aspects, the research has been specially oriented to Design, 
and Screen Design. Morrison et Alt. (1991) offer a revision about the research 
applications to the Design of CBI. They consider three aspects: the screen design, 
the control of the program, and the amount and type of feedback provided the 
learner. 

The research about the screen design is wide and deep. Works as these ones from 
William G. Sweeters (1985) and Madge, et Alt. (1986) summarized some of the 
more known results in this area. Or, from a more complex perspective of the 
interface, that is, including other elements as sound, the work of Lauran Sandals 

(1987) . Some aspects related with the screen design have to be reconsidered 
because the important changes in screen resolution and color, more the new 
photo and video realistic possibilities. 

The study of learner control has been studied by several colleagues, included my 
self in a previous work over Interactive video (Bartolome, 1992). Despite the 
results -see also Pridemore and Klein (1991)- developers seem not to be conscious 
of the problems related with the user control of the programme, specially at lower 
educational levels, perhaps because the need of "democratic environments" as 
described by Schwier and Misanchuk (1993). 

An interesting way has been the application of old results to new problems as in 
Collis (1991). The Multimedia has born in a context where some conclusions 
from other instructional media could be applied with some restrictions. 

Some special aspects related with the design of as-hypertext multimedia 
programmes have been studied as Shneiderman (1987) or Yankelovich et Alt. 

(1988) . A more open revision of this issue can be found in the well documented 
work of Jacob Nielsen (1990). 

In this short revision of some of the key subjects of the research that can be 
applied to educational multimedia programmes, we will end with the work over 
hardware. Different devices have been studied as the keyboard (Noyes, 1983; 
Litterick, 1981), Light pens (Whitefield, 1986), Mouse, Touch Panel (Karat et Alt. 
1986), etc. A more global perspective of the human-computer interface can be 
found in Barker (1989). 

The research in the application and design has been the key issue during the last 
years. But we asked for new trends in the research about multimedia production 
with more effective and practical results. An original idea was suggested by 
Hodges and Sasnett (1993) when said (p. 40): "Deriving from the philosopher 
Nelson Goodman's observations on art, the proper question should not be what 
is good design, butzohen is good design. For our purposes we can define good 
design as effective communication. When a message is communicated 
effectively, then there is good design... The design of an interface depends entirely 
on what it will be used for". 

But, as they recognized, that seems do not say a lot about how to produce 
multimedia programmes. But this idea includes a key element: the validity of the 
design is related with the use... and, that implies, with the user. So, in a 
development process, the role of the user could be important in order to achieve 
a success in the final product. 
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Which role? The end-users have not necessarily good skill in designing, drafting, 
painting, recording video,... and it is logic to suppose that they will be unfamiliar 
with the contents of the programme. So, it is clear that its role is situated 
primarily in the side of the evaluation and quality control. 

So, this is thr main objective of this work: to present how the end-users 
participate in the evaluation (can we use "quality control"?) of multimedia 
programmes in North America. 

2. Methodological Considerations 

The study runs over two stages. The first one is that we are going to present here 
and includes the answers to a questionnaire, and the analysis of some 
publications about the projects. The sample includes 26 programmes multimedia 
specifically referred to educational and instructional objectives. The data come 
from different people directly related with these programmes, usually involved 
in Coordination roles. 

These 26 programmes represent the work of more than 160 people during an 
average time of 2 years. This work costed more than $ 300,000 in direct expenses, 
more the human cost that could be estimated according to the average salary in 
between 2 and 3 millions of dollars. 

These programmes are actually 95'multimedia. Near every one began in the 
ninety's decade (figure 1) and the last versions ended between 1993 and the data 
collect time (figure 2). More than 80% of them were finished in 3 years or less and 
the most in 1 or 2 years (figure 3). 
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figure 1 




figure 2 
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figure 3 



The most of the programmes are oriented to Academic uses, and a few ones to 
Teacher Education (figure 4). So this sample is not representative of Industry 
multimedia programmes. That is a consequence of the objective of this work and 
of the sample selection process. This is coherent with the average cost (less than 
in Industry), the average development time (more than in Industry), and the 
used tools (authoring languages). 

Some projects were addressed to more than one level (Primary, Secondary...). 
This is an interesting characteristic in some Instructional programmes: it seems 
that words, pictures and graphics could be the same for a teenager, than for 
children or for 22 years old people, with different previous knowledge (e.g. 
vocabulary). 



Target group/s 




Primary 
Secondary 

University 

Industry/Mil. 
Teacher Education 



figure 4 



About the contents, these are oriented to Science and Technology (figure 5). In 
"Other" there are programmes with educational contents, History of Art, 
Management, Adult literacy. 
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figure 5 



The programmes were between first and fifth version (figure 6). Some of them 
were considered unfinished, and there were a general vision that a multimedia 
programme is never definitive. 
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figure 6 



About the Operative system, half and more have worked with Macintosh (figure 
7), with a percentage that does not differ from the other data (Barker, J., 1995). 
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This information is relevant, as the Authoring Language (table 1). Note thai 
sometimes are used more than one language. 
Table 1 



HyperCard 


8 


C 


6 


Director 


3 


ToolBook 


2 


Tencore 


2 


Common Lisp 


2 


Linkway 




IconAuthor 




Authorware 




Visual Basic 




Xwindow 




HyperStudio 




SmallTalk 




Other 





In around the 40 % the developers have used tools according to the "Hypertext" 
paradigm (HyperCard, ToolBook, Linkway,...). Certainly you must to consider the 
lower relative cost and the more open character about distribution licenses 
(HyperCard vs. Coursebuilder, toolbook vs. IconAuthor or AuthorWare...). Surely 
this is a difference between Academic and Industrial multimedia programmes 
(where e.g. Authorware is most used). 

Other aspect: in the 57% of the cases when the Macintosh computers are used, 
HyperCard is the selected language. MacroMind Director is the second one. About 
the "classic" programming languages, "C" has the most acceptation. 

How were these programmes? In the next point you will find some results. 
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Graphics, text, sounds™? 

The "text" is the winner!. In 52 % of the programmes, there is TEXT in EVERY 
SCREEN (figure 8), while in 44% of the programmes there are not VIDEO 
SEQUENCES NEVER. 




Never 



figure 8 

The use of these elements is conditioned by the characteristics and needs; so, it is 
reasonable that Animations and Video-sequences are used only sometimes (52% 
and 40%). But, if we study in how many programmes they are not used NEVER 
(figure 9) is clear that multimedia programmes have a big dependence from the 
written text. Sure, it is difficult to design a programme without text (but, is it 
impossible?), but 75% of the programmes had text in EVERY ONE or THE MOST 
of the screens!!!. 




figure 9 



The technical restrictions do not explain these results. Actually, if we apply 
contingency tables of this data with the Operative System, we do not find 
significant differences (0.4 < p < 0.5, according to the element). But jevery 
Macintosh has a built-in speaker, and the most a microphone! So, there is not 
significant relation between the introduction of sound in a multimedia 
programme and the capacity of the hardware. 
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We can find a justification in the production costs could, but it is difficult to 
consider that the Audio production is more expensive than Animations 
production. Other consideration could be related with the design of programmes 
oriented to be used individually in shared rooms. 

I have the feeling that the programmes reproduce an academic Communication 
conception, far away from the conception of the media (e.g. TV). 



The interface: from the man/woman to the machine 

The mouse is the prtferred interface (figure 10). The keyboard has today a 
secondary place. And Voice and VideoCapture (TO the computer) have a small 
role. At this moment, the most of the Macintosh includes microphone and its 
Operative System lets to accept instruction and to work with this device. It could 
be interesting to note the movement to these systems in the next years. Some 
developments in the field on AI will be critics for the introduction of these 
interfaces in the future. 
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figure 10 



Money, money, money™ 

This is a remarkable aspect because the high cost of multimedia production. 
However, in this case it is possible to find very "cheap" programmes: the 25% 
have needed less than $ 1,000. It is necessary to note that the personal costs were 
not included (figure 11). 
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figure 11 



How to recuperate this cost? Near the half of the programmes are being sold 
(figure 12). 
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This is more usual in the programmes with higher costs (Table 2). 
table 2 

Puiceats of Column Total* 





<1.0O0 


1.000-10.000 


10.000-20.000 


>20.000 


Totals: 


Yes 


25% 


40% 


33.33% 


71.43% 


47.37% 


No 


75% 


60% 


66.67% 


28.57% 


52.63% 


Totals: 


100% 


100% 


100% 


100% 


100% 



The Design of programmes 

It is possible to consider the instructional design of the programmes from 
different perspectives (Schwier and Misanchuk, 1993; Anglin, 1991). We used our 
own taxonomy, justified in other sites (Bartolome, 1994, 1995): In the domain of 
Information Presentation Programmes we include 3 models: Multimedia Book, 
Encyclopaedia and Hypermedia. In the domain of Learning Activities 
Programmes, other four models are showed: Practice, Tutorial, Problem Solving 
and Simulation. 

Actually, a programme could be included in more than one of these categories; 
however, it is possible to obtain some conclusions (Table 3). 

bible 3 

Learning Activities? 





Yes 


No 




Basically * es 


40% 


8% 


48% 


Information? ^ Q 


52% 


0% 


52% 




92% 


8% 


100% 



This table contains some kind of contradiction if you consider the text of the two 
questions: 

3.1 Does the programme include activities or questions for the user? (= the 
programme is not only a collection of Information items) 

3.3 Does the programme basically contain information that the user can retrieve? 
(== the programme does not contain learning activities as exercises, questions, 
problems, ...) 

So, the 92% include some activity or question, but only the 48% include learning 
activities as exercises, questions, problems... So, the most of the programmes do 
not include learning activities although they include some kind of feed back from 
the user (figure 13). 
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figure 13 

About the design, if we consider than Practice and Tutorial models are mostly 
based in Behaviorist theories, the Constructivist model is not a big reference for 
the design of multimedia programmes (17% + 4%) (figure 14). A similar result is 
obtained from the contents of Statistics courses in Education related with 
Qualitative vs. Quantitative methods: one thing is what people say, other what 
people use. 




About Information oriented programmes, the most of them are presented as 
Hypermedia (figure 15). The question is if they are actually hypermedia. I have 
found during the last years several developers that call "hypermedia" to every 
interactive multimedia Information programme. 
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figure 15 



Quality Indicators 
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We have tried of selecting some quality indicators: 

a) That there is a help system (despite the scarce use of it) 

b) The control of user over the programme 

c) The possibility of different levels for different users (that is, individual 
differences attention) 

d) Assessment (or Evaluation) system 
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e) Feed back to the user. 

The three last are the less considered aspects (figure 16): 





Doe* it include*...? 



10 20 30 40 50 60 70 80 90 100 



figure 16 



If you consider how many of these pointers appear in each programme, only in 
the 12% of them the number is less than 3 (figure 17). 




figure 17 



Production 



A third of the programmes were developed by 1 or 2 people. So, ^the multimedia 
programmes have to be actually the result of a team work? Is it possible to 
prepare multimedia as when a teacher prepare some notes for his/her students? 
(figure 18). 




figure 18 
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Of course, there is a significant relation between the number of people and the 
cost of the programme: the 100% of the projects with less than $ 1,000 costs have 
been produced by one or two people (Table 4). 

Table 4 
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Ptntafc of Row ToUk 

lew than 3 f3-6 (7-12 more 12 



<1000 


100% 


0% 


0% 


0% 


100% 


$1000-10000 


20% 


40% 


20% 


20% 


100% 


$10000-20... 


0% 


66.67% 


33.33% 


0% 


100% 


>20000 


12.5% 


25% 


37.5% 


25% 


100% 



Totals: 33.33% 28.57% 23 01% 1429% 100% 



Several authors consider that the script is the most relevant stage in the 
production of a multimedia programme in order to get the best results. We can 
consider that the number of people involved in the script is a quality indicator. In 
this case, the quality would be higher because only the 20% of the script are the 
work of one person (figure 19); the result is more significant if you consider that 
the 30% of the programmes have been produced with only 1 or 2 people. 




Assessment & Evaluation 

Two thirds of the programmes were evaluated during and at the end of the 
production process (figure 20). 



during 



at the end 




This is an interesting result and more considering that this evaluation or quality 
control is similar independently of the number of people involved in the project 
(TableS and 6). 
Table 5 
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Pcrerato of Row Totals 





Yea 


No 


Totals: 


!c*s than3 


62.5% 


37.5% 


100% 


0-6 


80% 


20% 


100% 


(7-12 


66.67% 


33.33% 


100% 


more 12 


66.67% 


33.33% 


100% 


Totals: 


68.18% 


31.82% 


100% 



Table 6 



Percent* of Row Totals 





Yes 


No 


Totals: 


ss than 3 


50% 


50% 


100% 


f3-6 


71 .43% 


28.57% 


100% 


(7-12 


71.43% 


28.57% 


100% 


more 12 


66.67% 


33.33% 


100% 


Totals: 


64% 


36% 


100% 



If we consider the two kind of evaluation, the same projects that include one of 
them, they include the other (p=0,001). 

Speaking of the evaluation "during" the production process, we took in 
consideration WHEN. In more than 50% of projects, the evaluation takes place in 
2 moments, more the end control. Although there is not a clear taxonomy, we 
have found these profiles as more usual ones: 

a) Who does not assess the programme never 

b) Who assess the programme one time, during or at the end or process. 

c) Who assess the computer software and, later, the final product. 

d) Who assess the results in several stages of the production. 

And these are the corresponding percentages: 
profile A 25 % 
profile B 15% 
profile C 15% 
profile D 45% 

End user participation 

End users participated in the production of the project: two thirds. But less than 
50% in the case of "evaluation during" (figures 21 and 22). 
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figure 21 




figure 22 



The quality of the product 



The first indicator of the quality of the programme is the self-perception of the 
authors (figure 23). 



How do you define the overall quality of the product? 




figure 23 



Good!. Well, we are going to be more specific: What about the educational 
objectives? Do you get them? (figure 24). 
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figure 24 

These results are obtained also from these projects where there was not any 
evaluation during or at the end. It sounds the cited revision of Bosco (1986). 



The more or less quality of a programme is coherent with the distribution level: 
the most copies distributed the most positive the evaluation of the programme 
(table 7). - 
Table 7 



Very good Good 



less 10 .4 7 11 



flO- 100 0 3 3 



more 100 7 2 9 



Totals: n 12 23 



A similar result related with the achievement of educational objectives (Table 8). 
In both cases the p is near of 0.01. 

Table 8 





Yes 


Most 


Only part 




less 10 


5 


5 


1 


11 


flO- 100 


0 


3 


0 


3 


more 100 


9 


0 


0 


9 


Totals: 


14 


8 


1 


23 



Feed-back from end users 

In the most of the cases, the opinion of end users has been asked (figure 25). 
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Some feed back from actual uien? 




4. Discussion 

At this moment, new contacts with interviews had to take place with the 
developers. But we can to obtain some initial results. 

These are the aspects that I have selected, but surely other people can find other 
more relevant from their point of view. 

1. The educational programmes are in most cases (50%) "Information distribution 
. programmes" 

2. The "hypermedia" model is widely suited. 

3. The continuous evaluation during the production of the programme, and the 
participation of end users in this evaluation, they both are notorious but they are 
not generalized. 

4. Objective mechanism to control the quality of programmes is not 
systematically used. 

5. The multimedia programmes have a great dependence from the text (perhaps 
because the kind of contents or instructional objectives). 

6. The mouse is the most common interface for the user. (But, is it the more 
adequate in these programmes?) 

7. The multimedia conception of communication is being used at a high and 
costly level (developers), as at a low level (professors), thanks to the authoring 
tools. And because the no consideration of the personal costs, the differences 
between budgets have not relation with the quality of results. 
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